Speech Pen: New Pen Input Interface Capable of Utilizing Speech Recognition for Digital Writing

This paper introduces a multimodal input system, called “speech pen” that assists digital writing during lectures or presentations with background speech and handwriting recognition. The instructor basically freely speaks to the audience and writes on an electronic whiteboard as usual. The system recognizes those speech and handwriting in the background and provides the instructor with predictions for the further writing by using the recognition results. The instructor can accept a prediction and paste it in the board to save manual writing. If all predictions are wrong or useless, the instructor can simply ignore them. The speech-pen system also allows the sharing of context information for predictions among the instructor and the audience; the speech recognition result of the instructor is sent to the audience to support their own note taking. A preliminary study shows the effectiveness of this system and the implications for further improvements.